Papers by Deven Mahesh Mistry
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that attention heads have a temporal induction property that allows them to learn and reproduce sequences of tokens. |
| Approach: | They analyze attention heads and transformer outputs to examine in-context temporal biases . they find that transformer output has a tendency toward in-constext serial recall . |
| Outcome: | The findings shed light on similarities and differences between LLMs and human memory and learning. |
Beyond Semantics: How Temporal Biases Shapes Retrieval in Transformer and State-Space Models (2026.eacl-long)
Copied to clipboard
| Challenge: | In-context learning depends on what appears in the prompt and on when it appears. |
| Approach: | They construct prompts with repeated anchor tokens and average their predictions over hundreds of random permutations of the intervening context. |
| Outcome: | The proposed model predicts the same tokens over hundreds of permutations over time. |